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A distributed network management function is implemented 
in a computer network using a set of active nodes. Each of 
the active nodes comprises a router and a logically-separate 
active engine. The router in a given one of the active nodes 
diverts active packets associated with the network manage- 
ment function to the corresponding active engine for pro- 
cessing. The active engine supports one or more sessions, 
based at least in part on the active packets, for implementing 
at least a portion of the network management function. Each 
of the sessions supported by the active engine corresponds 
to a particular distributed task to be performed in the 
network, and has associated therewith a unique network 
identifier, such that different programs on different network 
nodes can belong to the same session. The router and active 
engine at a given one of the nodes may reside on the same 
machine, or on physically -separate machines. 
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FIG. 4 



tlshrey 268 act> traceroute shvat 

traceroute to shvat (192.180.140.203), 30 hops max, 40 byte packets 

1 heshvan (135.180.142.2) 0.348 ms 0.236 ms 0.219 ms 

2 kislev (135.180.142.10) 0.708 ms 0.624 ms 0.599 ms 

3 razciscol (135.180.142.18) 2.620 ms 382.160 ms 7.236 ms 

4 shvat (192.180.140.203) 160.888 ms 4.052 ms 4.196 ms 



FIG. 5 
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FIG. 6 



hop 1: Object ID : .1.3.6.1.2.1.1.1.0 (Syntax : Octet) 

Value : FreeBSD tishrey.dnrc.bell-labs.com 2.2.2-RELEASE FreeBSD 2.2.2-RELEASE #0 

: Mon Feb 23 04:01:12 EST 1998 root@tishrey.dnrc.b 

hop 2: Object ID : .1.3.6.1.2.1.1.1.0 (Syntax : Octet) 

Value : FreeBSD heshvan.dnrc.bell-labs.com 2.2.2-RELEASE FreeBSD 2.2.2-RELEASE fO 

: Mon Jun 8 13:15:35 EST 1998 root@heshvan.dnrc.b 

hop 3: Object ID : .1.3.6.1.2.1.1.1.0 (Syntax : Octet) 

Value : FreeBSD kislev.dnrc.bell-labs.com 2.2,2-RELEASE FreeBSD 2.2.2-RELEASE #0 

: Thu Dec 3 05:04:35 EST 1998 rootikislev.dnrc.be! 

hop 4: Object ID : .1.3.6.1.2.1.1.1.0 (Syntax : Octet) 

Value : Cisco Internetwork Operating System Software 

IOS (tm) 2500 Software (C2500-I-L), Version 11.3(3), RELEASE SOFTWARE (fcl) 

Copyright (c) 1986-1998 by Cisco Systems, Inc. 

Compiled Mon 20-Apr-98 18:23 by phanguye 

hop 5: Object ID : .1.3.6.1.2.1.1.1.0 (Syntax : Octet) 

Value : FreeBSD shvatdnrc.bell-labs.com 2.2.2-RELEASE FreeBSD 2.2.2-RELEASE #0 

: Wed Dec 23 04:27:46 EST 1998 root@shvatdnrc.bell- 
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FIG. 7 

import Act*; 
import OurSnmp.*; 
public class capsule 

i 

public static void main ( String args[] ) throws Exception \ 

DatagramPacket udppacket; 
Act session = new Act(-9); 
byte[] p = session.getProg(); 
byte[] v = sesslon.getlnitVarsQ; 
byte[] destip = new byte[4]; 
byte[] udpmsg; 

// get target IP address 

for (int i=0; i<4; destip[i] = v[i+4]; 

// get hop number 

int hopnum = (int) v[8j; 

if (v[8]>127) System.outprintln("too big "); 

else v[8]++; 

// prepare a new message 

byteQ newpck = new byte[pJength+9]; 

for (int i=0; i<p.length; i++) newpck[i] = p[i]; 

for (int i=0; i<9; i++) newpck[i+p.length] = v[i]; 

// send a new message forward 
sesslon.send(newpck l AciIPaddr(destip)); 

// get some local status (via SNMP) 

String oid = ".1,3.6.1.2.1.1.5.0"; //host name 

String resl = OurSnmp.Get(oid); 

// send a UDP datagram to report your status 
String udpmsgtext = "hop " + v[8] + ": " + resl; 
session.sendUDP(udpmsgtext, "inbar.dnrc.belhlabs.com", ReportPortNum); 

//be nice, report you are done. 
session.killme(); 

i 

j 
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inbantry> java UDP 
hop 1: Object ID : 
Value : 1178310 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 


hop 2: Object ID : 
Value : 491 BoZ 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 


hop 3: Object ID : 
Value : 213 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 


hop 4: Object ID : 
Value : 1066733 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 


hop 5: Object ID : 
Value : 0 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 


hop 1: Object ID : 
Value : 1178361 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 


hop 2: Object ID : 
Value : 491911 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 


hop 3: Object ID : 
Value : 215 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 


hop 4: Object ID : 
Value : 1066771 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 


hop 5: Object ID : 
Value : 0 


.1.3.6.1.2.1.4.6.0 


(Syntax 


: Counter32) 



FIG. 9 
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lYiina nvju /\mj aitakai ua rwK 
EFFICIENT NETWORK MANAGEMENT 
USING AN ACTIVE NETWORK 
MECHANISM 

FIELD OF THE INVENTION 

The present invention relates generally to computer net- 
works and more particularly to network management tech- 
niques for use in computer networks. 

BACKGROUND OF THE INVENTION 

The emerging next generation of routers for computer 
networks exhibit not only high performance, but also 
enhanced functionality, such as support for virtual private 
networks and quality of service (QoS) guarantees. In order 
to achieve this functionality, features such as per flow 
queuing and fast Internet Protocol (IP) filtering are generally 
incorporated into the router hardware. However, the man- 
agement of a network comprised of such devices and 
efficient use of the enhanced functionality introduces many 
new challenges which are not adequately addressed by 
existing network management techniques. 

The majority of conventional network management sys- 
tems are typically centralized around some type of manage- 
ment station. In this type of centralized architecture, a 
manager queries the managed objects, builds a view of the 
network, and sends alerts if a problem is detected. The 
manager can also try and take corrective actions by sending 
configuration commands to network entities. 

There are many drawbacks to the above -noted centralized 
architecture, and these drawbacks generally become more 
evident as a network grows in size and complexity. For 
example, as the number of controlled elements increases, the 
requirements for computational power from the manage- 
ment system and bandwidth from the network that connects 
it also increase. In addition, in a large network, some of the 
controlled entities are distant from the management station, 
such that control loops exhibit long delays, and control 
traffic wastes bigger portions of the network bandwidth. 

In order to alleviate the above -noted scalability problems, 
various types of distributed control architectures nave been 
proposed in recent years. See, e.g., G. Goldszmidt et al,, 
"Distributed Management by Delegation," 15th Interna- 
tional Conference on Distributed Computing Systems," 
IEEE Computer Society, Vancouver, Canada, June 1995; Y. 
Yemini et al., "Towards programmable networks/' IFIP/ 
IEEE Intl. Workshop on Distributed Systems Operations and 
Management, October 1996; A. A. Lazar, "Programming 
telecommunication networks/' IEEE Network, 11 (5): 8-18, 
September/October 1997; and M. Zapf et al., "Decentralised 
snmp management with mobile agents/ 1 Sixth IFIP/IEEE 
International Symposium on Integrated Network 
Management, IM'99, May 1999, Boston, Mass. 

Most of these conventional approaches delegate some of 
the central management tasks to distributed software agents. 
This is very useful to alleviate the load from the manager, 
reduce network traffic, and shorten control loops. However, 
this type of delegation does not support a truly distributed 
agent system where agents can communicate with their 
neighbors to efficiently carry out distributed tasks. Such 
distributed tasks are important in implementing functions 
such as scalable self-healing in network operations, e.g., 
local rerouting around congested spots, or cooperated cach- 
ing. 

Many of the above-noted conventional approaches also 
use a distributed object paradigm that abstracts the imple- 



mentation details. Abstractions, such as Common Object 
Request Broker Architecture (CORBA), Distributed Com- 
ponent Object Mode land (DCOM), Java Remote Method 
Invocation (RMI), are helpful in designing and building 

5 distributed agent systems, but they hide the true cost of the 
implementation details. As a result, conventional agent- 
based systems tend to be, in many cases, inefficient in their 
use of network resources, primarily in their use of band- 
width. These and other conventional network management 

io approaches thus fail to provide efficient distribution of the 
management task in the network. 

Another important issue in distributed agent systems for 
management and control is the location at which these 
agents reside. Clearly, this has a major impact on the 

15 performance since it effects the delay in the control loop. In 
general, existing software agent solutions assume the exist- 
ence of available hosts to run their application-level pro- 
grams. An optimal location for an agent would be in the 
router kernel where all the necessary local information is 

20 available, and action can be taken locally. However, such a 
solution has been deemed impractical due to the inability to 
interfere with router real-time operation constraints. Clearly, 
the closer the agent is to the controlled system, the better it 
can perform. Nonetheless, conventional distributed network 

25 management approaches have been unable to provide opti- 
mal agent location. 

It is therefore apparent that a need exists for an improved 
distributed network management approach which over- 
comes the problems associated with the above-described 

30 conventional approaches. 

SUMMARY OF THE INVENTION 

The invention provides an active network approach to 
network management that simplifies the deployment of 

35 distributed network management applications in IP networks 
and other types of computer networks. In an illustrative 
embodiment of the invention, a distributed network man- 
agement function is implemented in a computer network 
using a set of active nodes. Each of the active nodes 

40 comprises a router and a logically-separate active engine. In 
addition to performing conventional forwarding functions, 
the router in a given one of the active nodes diverts active 
packets associated with the network management function to 
the corresponding active engine for processing. The active 

45 engine supports one or more sessions, based at least in part 
on the active packets, for implementing at least a portion of 
the network management function. Each of the sessions 
supported by the active engine corresponds to a particular 
distributed task to be performed in the network, and has 

50 associated therewith a unique network identifier, such that 
different programs on different network nodes can belong to 
the same session. The router and active engine at a given one 
of the nodes may reside on the same machine, or on 
physically-separate machines. 

55 In accordance with another aspect of the invention, the 
given active node may be configured to support a blind 
addressing mode. The blind addressing mode may be used to 
send designated packets from the active node via unspecified 
network nodes in a particular direction toward a destination 

60 node, such that a router in a first unspecified active node 
along a route to the destination node intercepts the packet 
and sends it to an associated processing engine of that node. 
In this manner, the active node can communicate with other 
active nodes of the network without knowing their specific 

65 addresses or locations. 

Examples of network management functions that may be 
implemented using the active network approach of the 



04/01/2004, EAST Version: 1.4.1 



US 6,529,515 Bl 

3 4 

invention include coiiect-en -route and report-en-route infor- network techniques to distribute and execute network man- 
raation gathering programs, and message dissemination agement applications, the invention provides efficient use of 
applications with ad-hoc definition of a particular group of network resources, without increasing the complexity of 
message receivers. application development. Advantageously, a computer net- 
By utilizing active network techniques to distribute and 5 w ° r k configured in accordance with the invention enables 
execute network management applications, the invention the safe execution and rapid deployment of new distributed 
provides efficient use of network resources, without increas- management applications in a network layer. This active 
ing the complexity of application development. network approach can be gradually integrated into, e.g., an 
Advantageously, a computer network configured in accor- otherwise conventional IP network, and allows smooth 
dance with the invention enables the safe execution and 10 migration from conventional IP to programmable networks, 
rapid deployment of new distributed management applica- FIG. 1 is a block diagram of a network node 10 in the 
tions in a network layer. This active network approach can architecture of the illustrative embodiment. This embodi- 
be gradually integrated into, e.g., an otherwise conventional ment will be described using active network terminology. 
IP network, and allows smooth migration from conventional More particularly, network node 10 will be referred to herein 
IP to programmable networks, 15 as an active node, packets that carry agent code and com- 
munications among agents will be referred to as active 
BRIEF DESCRIPTION OF THE DRAWINGS packets, and the corresponding traffic will be referred to as 

~^ ^ t .j- /• j ■ * *r a j active traffic. In the figure, thick lines between components 

FIG. 1 shows an Mutative embodiment of a d>stribu ed ^ a , e fl * w Qf ^ ^ ^ ^ a 

network management arch.tecture ,n accordance with the ^ ^ coni f eclion It should ^ understood lhal t £ e active 

invention. n(xle ^ ^ g enera ny p ar t 0 f a larger network that includes 

FIG. 2 shows the structure of a default option in an active multiple interconnected nodes, only a subset of which need 

network encapsulation protocol (ANEP) header used in the ^ act j ve noc jes 

illustrative embodiment of FIG. 1. ^ ac(ive Qode 1Q Qf FJG x a M u and 

FIG. 3 is a block diagram of a simple network used to 25 an acliye engine 14 ^ router 12 is coup i ed t0 a commu . 

illustrate the operation of the network management archi- n icatton network via network connections 15 established 

tecture of FIG. 1. through ports 16. The router 12 includes a routing processor 

FIG. 4 shows an example of a route tracing execution 17, a simple network management protocol (SNMP) module 

from a particular one of the hosts in the FIG. 3 network, 18, and a diverter 19. The router 12 performs conventional 

FIGS. 5(A), 5(B) and 5(C) illustrate three different route 30 IP routing functions such as forwarding, routing, and filter- 
tracing executions that may be implemented in the FIG. 3 ing. The diverter 19 detects and diverts active packets to the 
network. active engine 14, e.g., if the protocol type and port number 

FIG. 6 shows an example of a router identifier report °J the P ackets ™*fa a designated mask. The diverter 

generated in accordance with the invention. 35 ma y b u e implemented ,n hardware or software or 

_ , , , - i , * . combinations thereof, depending on the particular unple- 

FIG. 7 shows an example set of active packet code that - . , 1 I ° r 

, ... ,i , J , lU ■ mentation of the router 12. 

implements data collection along a particular path in a . . . 

network in accordance with the invention. T»» a « ive en S ine 14 * an entltv . 15 lo S lcaU y 

0 , t - * i separate from the router 12 and which performs certain 

FIG. 8 shows an example of a report generated by an J [[yQ ioas of me aclive Q F ode 10 It 

active data collection program for an IP forwarding counter 40 ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 

in accordance with the invention. n of oq a ^ a 

FIG. 9 shows an example of a binary balanced dissemi- conventional off-the-shelf IP router to be upgraded to an 

nation tree utilized in a message dissemination application active router s i mp i y by adding an adjunct active engine, 

of the invention. ^ Moreover, the separation of the router 12 from the active 

TTCn nccpDTBTrmT nc tue engine 14 protects non-active traffic from the effects of 

DETAILED DESC^TION OF THE erroneous operation of the active part of the network, while 

INVEN HON imposing only minimal additional delay on non-active traf- 

The present invention will be illustrated below in con- fic. It also simplifies gradual deployment of active nodes in 

junction with an exemplary computer communication net- 50 casting networks. 

work configured to utilize the Internet Protocol (IP) stan- The active engine 14 includes an active manager 20, and 

dard. It should be understood, however, that the invention is supports N sessions 22-1, 22-2, . . . 22-N. Portion 24 of the 

not limited to use with any particular type of network or active engine 14 identifies the elements associated with the 

network communication protocol. The disclosed techniques active engine kernel. These elements include an IP connec- 

are suitable for use with a wide variety of other networks and 55 tion with the router 12, and a security stream module 25 for 

protocols. providing security functions to be described in greater detail 

The invention in an illustrative embodiment provides a below - The active engine 14 may be viewed as an execution 

computer network architecture that simplifies the deploy- environment in which user-written application program 

ment of distributed network management applications in IP codc » encapsulated in active packets, can be executed with 

networks and other types of computer networks. As will be 60 close interaction with router data and control variables. An 

described in greater detail below, the invention makes use of example of an execution environment is given in, e.g., AN 

so-called active network techniques, as described in, e.g., D. Working Group, "Architectural framework for active 

L. Tennenhouse et al., "A survey of aclive network networks," Version 0.9, http://www.cc.gatech.edu/projects/ 

research," IEEE Communications Magazine, 35(l):80-86, canes/arch/arch-0-9.ps, Aug. 31, 1998. 

January 1997, to provide a framework which allows the 65 Each of the sessions 22-1, 22-2, . . . 22-N represents a 

efficient distribution and execution of network management distributed task performed in the network. Each session has 

applications in a set of network routers. By utilizing active a unique network identifier, such that different programs on 
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various nodes can belong to the sanic session. TlieSc piu- authentication, creation of a control block for the session! 

grams may exchange information using active data packets, creation of a protected directory to store session files, 

and can distribute and/or update their code by sending active opening of a private communication channel through which 

programs. The session identifier allows a session to serve as the session receives and sends active packets, and execution 

a rendezvous point for data from different agents or other 5 of the code. 

programs, and can support code upgrades "on-the-fly." It The above-described sessions allow the corresponding 

also supports a so-called "one-shot" capsule model by programs to easily send themselves to another node, and to 

allowing a particular program to signal its termination, and send and receive data. New arriving programs are passed to 

to trigger a clean-up. the sessions to allow them to perform code updates without 

The active manager 20 generates the sessions, coordinates to losing state, 

the data transfer to and from the sessions, and "cleans up" The illustrative embodiment includes both a blind 

after a session when that section terminates. While a session addressing mode and an explicit addressing mode. A distinct 

is alive, the active manager monitors the session resource UDPport number is assigned in this embodiment to each of 

usage, and can decide to terminate the operation of a given these modes. The first UDP port number (3322) is the blind 

session if that session consumes too much resources, e.g, 15 addressing port, which is used to send active packets to 

CPU time, memory, bandwidth, etc., or if it tries to violate unspecified nodes in a certain direction, i.e., towards some 

its action permissions, distant destination. The diverter in the first active node that 

The security stream module 25 resides in the active is on the route to that destination intercepts the packet and 

engine kernel below the IP output routine. Every connection sends it to the active engine of that node. The sender is 

that a given session wishes to open must be registered with 20 therefore not required to know the address of the next active 

this module in order to allow monitoring of network usage node. This allows an active node to communicate with other 

by sessions. The registration may be performed in a manner active nodes in order to perform distributed tasks, without 

transparent to the application developer. storing fall network topology information at each node. The 

The SNMP module 18 is an SNMP agent implemented in blind addressing mode is particularly useful for functions 

the router 12. This agent serves as an interface between the 25 such as topology learning, robust operation, support of 

router 12 and the active engine 14, allowing the sessions heterogeneous environments, etc. The second UDP port 

22-1, 22-2, . . . 22-N of the active engine 14 to access a number (3323) is an explicit active port, which is used to 

managed information base (MIB) of the router 12. The t send an active packet to a specific active node. This packet 

MIB may include network layer data of the router, e.g., is forwarded directly through all of the intermediate active 

topological data such as neighbor identifiers, routing data, 30 nodes of the path, and is not diverted to an active engine 

performance data such as packets dropped, packets until reaching its destination node. 

forwarded, CPU usage, etc. The SNMP API in the sessions A session that requires non-active communication with 

22-1, 22-2, . . . 22-N may be implemented as a Java object other elements of the network, such as communication using 

that communicates with the router 12 using well-known SNMP, HTTP, etc., must utilize an appropriate interface to 

SNMP techniques. Standard SNMP agents exist in most 35 the relevant Java objects. These interfaces transparently 

conventional routers and provide a read/write interface to a register the communication channel in the security stream 

standard MIB. Id other embodiments, performance may be module 25. An attempt to bypass these registrations will 

enhanced by caching popular MIB objects. result in the security module 25 dropping the packets of the 

Although the illustrative embodiment allows multiple 4Q unregistered connection, 

languages to be implemented simultaneously, the following The active manager 20 monitors the resource consump- 

description will focus on an implementation for handling tion of the sessions in the node. This monitoring is done to 

Java packets. Implementation of other languages may avoid a situation in which a session takes a large portion of 

require some straightforward adaptation according to the the system resources, due to an error or malicious intent. For 

language specifics, as will be apparent to those skilled in the 45 this reason, the monitoring is done on a per-node basis and 

art. not globally. A session that consumes excessive resources is 

The flow of packets through a network including the aborted, 

active node 10 will now be described in greater detail. Note The active manager 20 may abort a session in at least two 

that a non-active packet does not pass through the active other situations. The first is when the session does not show 

engine 14 since the diverter 19 recognizes it as such and thus 50 activity within a specified aging period. Different sessions 

the packet moves directly to its appropriate output port on may have varying aging periods, but none will be able to 

the router 12. , reside dormant in the CPU indefinitely. This allows "garbage 

All active packets in the illustrative embodiment include collection" of unused code. A session might also ask to be 

a default option that contains the unique session identifier of aborted gracefully. The active manager 20 allows a graceful 

the packet, a content description, e.g., data, language, etc. 55 clean up of all the objects associated with such a session, 

FIG. 2 shows an example of the structure of the default especially packets that are waiting for transmission in the 

option in an ANEP header in the illustrative embodiment. In node. 

this example, version is the software version, and lang is the Since it is expected in many applications that network 

language identifier. Only the most significant bit in the flags programming will be relatively stable, one may be less 

field is currently assigned to identify the last segment. The 60 concerned about program size since programs may not need 

segment number is given by seg, and ses_seq and ses_id to be transmitted frequently. The illustrative embodiment 

comprise the session identifier. therefore does not attempt to optimize the capsule model. 

In operation, all the packets diverted by the diverter 19 to The active engine in this embodiment instead includes a 

the active engine 14 are sent to the active manager 20. If a mechanism to reassemble a program from a chain of up to 

packet does not belong to an existing session and it contains 65 2 56 UDP packets. 

code, it triggers a creation of a session. If it is a data packet, The modular architecture illustrated in FIG. 1 supports 

it is discarded. A session creation may involve, e.g., interoperability, and does not require that the specific 
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The uiverter module cvcui uf a crash of ihe active engine, h is also important in 
19 can be implemented using, e.g., IP filtering which is providing safety to ensure that a session will not corrupt or 
supported in the API level in most conventional routers. This even access data associated with other sessions. This may be 
architecture also allows an easy incremental deployment in achieved in the illustrative embodiment through the use of a 
heterogeneous networks. Another advantage of this archi- 5 program such as Java Security Manager, which may be used 
lecture is robustness, i.e., non-active traffic is not affected by t0 con trol the session running environment, and more par- 
active traffic. More particularly, even if for some reason the ticularly to prevent sessions from using native methods and 
active engine stops working, the router will still route tQ restrict the use of me fiJe tem 

non-active packets correctly. _ . . , „ 

...... i u * . ii . Security implies that all operations including access to 

It is apparent that the FIG. 1 architecture allows one to in , . ' r ,. , , . r . . , to . 

rK . ... . ,. . . . . , 10 data are authenticated, i.e., only authorized sessions can 

simultaneously run multiple distributed network manage- - .. , 4 , A „, ... t . 

,. , . c , . .r j pertorm actions and or access private data. The illustrative 

meat applications. A given one of the above-described r , , „ u . , . r . r 

. rr , ii r 4 *i_ , embodiment allows multiple levels of security via authen- 

sessions may be a collection of agents or other programs that , . T_ . ' n , 

. . . 5_ . j j < j tication and session classification. More specifically, each 

are miected to the network by authonzed users and executed 4 , . . . J 

J . rTTi t -*r j t session is authorized to use specific services such as, e.g. 

in the active engines. The agents can migrate trom node to «<- ., Tri - . i * » 

, , j i. . iL i a T i_ i * iL 15 Mi B access for re ad or wnte, divert non-active packets, etc.. 
node and can duplicate themselves. Agents belonging to the , , ' T . t . , , . J\ 

. t . ^ ♦ .u • i u ancl resources such as, e.g., CPU time, bandwidth, memory, 

same session in different nodes can cooperate their work by ^ Tq ^ en ^ of ^ of netwQrk ^ 

exchanging data messages. Using a well-defined network . . , , ~ . . . . 

. . r * w IT , ? . , , iU resources by sessions may be monitored. For example, tight 

layer interface, i.e., an MIB, authorized agents can gather „ , / . . i . .u 

. ' , . c , / , r j * control over the usage of communication channels to other 

local information and control the lorwarding operation. ~ n , , , ~i * -rr>n ** ^ j 

_ . - i_ ■ t * j 20 outside-network elements via TCP connections is provided 

Other software distribution mechanisms can be integrated as . • .u . l. *■ ui JL 

i * j * u by ensuring that such connections can be only opened by a 

well, e.g., popular network management programs can be j • • .u j , l . ■« *i_ i. j -j.t. 

• r - i -. t . * ^ r . L • permitted session using methods that monitor the bandwidth 

grouped into libraries that are made part of or otherwise r A „ ° 4 , , . , 

• . j • consumption. An attempt to use conventional Java methods 

assoaated with the active engine. may be blocked by controlling the IP layer in the active 

As prevously noted, the architecture in the illustrative 25 engine. An unauthorized connection will be dropped. UDP 
embodiment utilizes an active network type of framework, kets can ^ sent Qn , th h thc acliye man whid) 

i.e., a framework where network elements primarily routers in caQ monitor (hc bandwidth 

and switches, are programmable, as described in, e.g., D. L. _ ¥ _ . ( , . . 

Tennenhouse et al., "A survey of active network research," , 3 shows an «« n I^. ™plementation of a small 

IEEE Communications Magazine, 35(l):80-86, January 30 heterogeneous nework 30, wh.ch illustrates the operauon of 

1997. Programs that are injected into the network are J he u above-desenbed architecture. The network 30 estab- 

executed by the network elements to achieve higher flex- llst J es connection between the Internet 32 and six active 

ibility for networking functions, such as routing, and to nodes ^ ac ' lve nodes '° dude routers ^ 34 ^ 

present new capabilities for higher layer functions by allow- «sM™tod with corresponding active engines 35-1 through 

. , , e . . _ ii, 35-6, respectively. All router-engine pairs other than route r- 

lne data fusion in the network layer. 35 ,* ^ . 

. . , . . , . engine pair J4-4, 35-4 are implemented on common 
Additional details regarding active network techniques i_* n * ia c j ia / * c 1 1 

, . c . . 0 . ° . . tU , . machmes. Routers 34-5 and 34-6 are part or a local area 

suitable for use in coniunction with the present invention are t . ™ t . . r t . , 

described in ee D Wetherall et al "ANTS' A toolkit for netWOrk 3? * ^ r0UterS 34 " J 10 tlm netWOrk may be 

. u* ^ j 11 j 1 • -1 . 1 ». implemented using, e.g., otherwise conventional FreeBSD- 

buildine and dynamically deploying network protocols, , r , .. * 1Cftn . . r 4 ^ . 

nMnvrfnoM.^ , i-^ 1™ a -i ; nno A n v 11 • . based active routers, Cisco 2500 routers and Lucent Tech- 
OPENARCH 98, pp. 117-129, April 1998; A. B.Kulkarniet 40 1 nADim Xa * 1 .u ui 

«. K f . . ' • . , „ nologies RABUPortMaster3 routers, or other suitable rout- 

al., "Implementation of a prototype active network, * FreeBSD-based routers mav be implemented as 

OPENARCH J 98, pp. 130-143, April 1998; D. Scott Alex- erS> ih * breeB f U based routers ^ be im P lem |° ted as 

, «™ o . .vi 7 T . 1 ,v „ personal computers runmng the well-known FreeBSD oper- 

ander et al., The Switch Ware active network architecture, \. 4 %_ t . 0 . . r .. 

Trrn xr ' . ^/jx^no^ 1 1 ik * mno c t»u * ating system. The active engines 35-/ are wntten primarily 

IEEE Network, 12(4):29-36, July/August 1998; S. Bhat- . J r A . , - a *u • 1 <? 

... 1 « * . . ' 1 ■ « inC code, and Java is used as the programming language for 

tachaiiee et al., An architecture for active networking, 4s , nu u *u * .* r ° u 

TmvT1 i n . - * • «a r 1 the active code, although the invention can of course be 

HPN 97, April 1997; E. Amir, "An active service framework , , , . . t tU P tU 1 _ 

. ' V. . ' ' . . j . j • implemented in and support the use of other known pro- 

and its application to real-time multimedia transcoding, r . t 

SIGCOMM'98, September 1998; Y. Yemini et al., "Towards jamming languages. 

programmable networks," IFIP/IEEE Intl. Workshop on ^ divcrter function in the routers 34-; is implemented 

Distributed Systems Operations and Management, October 50 Wlth fillers accordin S to the active P ort identifier. In the 

1996; D. Decasper et al., "DAN: Distributed code caching FreeBSD routers, active packets may be diverted to the 

for active network," INFOCOM'98, March 1998; and D. active ™&™ b V FreeBSD packet filter software, ipfw. In the 

Scofif Alexander et al, "The active network encapsulation cisco router ' the Averting may be performed using Cisco 

protocol (ANEP)," http://www.cis.upenn.edu/-switchware/ Internetwork Operating System (IOS) features. In the 

ANEP/docs/ANEPtxt 1997 55 Lucent PortMaster router, the diverting may be performed 

Safety and security are major concerns in incorporating a usin £ a tunnelin fi fea * ure * M previously noted, the invention 

distributed control mechanism in a computer network. can be used with other types of routers. 
Safety implies that no application can destroy or damage the ^ should be noted that the particular configuration of 

appropriate execution of other applications. In particular, the network 30 as shown in FIG. 3 is an example for illustrative 

active engine as a whole should not effect the routing of 60 purposes only, and is in no way intended to limit the type of 

non- active packets. In the architecture of the illustrative network configurations in which the invention can be imple- 

embodiment, safety is achieved by logically separating the mented. 

execution environment from the forwarding mechanism, and The manner in which the architecture of FIG. 1 can be 

by using a well-defined interface between them. In an used to create efficient distributed network management 

embodiment in which the active engine resides on a separate 65 applications will now be described in greater detail. T\vo 

machine, such that the separation not only logical but exemplary applications will be described. The first applica- 

physical, the non -active data will not be affected even in the tion is bottleneck detection, which is a special case of 
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coilcuiiug aiiumituiuij ui calculating a function along a route FIG. 5(A). This is because a message to a node at distance 
between two nodes. The second application is message i is sent only after a reply from node at distance i-1 has been 
dissemination for a large group of receivers. It is useful for received. The active network approach of the present inven- 
automatic configuration of network elements or any other tion reduces the time complexity because actions can be 
application that requires dissemination of messages to a 5 taken in intermediate nodes based on local information. As 
large population. a result, both the collect-en-route and report-en-route pro- 

Bottleneck detection is an important problem in network grams have a linear time complexity. The difference between 
management. It is a building block for higher level them is that in collect-en-route all the information arrives to 
applications, e.g., video conferencing, that require QoS lne source together, while in report-en -route partial results 
routing. It also serves as an example of a problem related to 10 are available sooner, but at the cost of an increase in message 
gathering of information along a given path between two complexity. 

network nodes. It should be noted that the above-described programs can 

Conventional IP networks generally provide only a single be configured to collect any desired data from the router, 
ad-hoc technique to examine one specific QoS parameter, rather than just a router IP address. For example, an appli- 
i.e., the delay along a path. This technique is the well-known 15 cation programmer can query any available variable, such as 
trace route program that enables a user at a given host to get an MIB variable, associated with the router. As another 
a list of all the routers on the route to another host with the example, in a bottleneck detection application, statistics 
elapsed time to reach them. FIG. 4 shows an example of a aD °ut TCP packet loss along a route to a certain host can be 
traceroute execution from the host tishrey in the network of collected in order to identify the bottleneck link. In addition, 
FIG. 3. However, the use of the traceroute program for 20 the programs can be further generalized to allow a node to 
network management has a number of significant draw- perform the data collection on the path between any other 
backs. For example, it can only retrieve the hostname and two active nodes in tne network. This is facilitated by the 
the delay along a path, it is extremely inefficient in its use of two addressing modes described above. As mentioned 
network resources, and it is slow. previously, the resulting reports can be sent to any host, and 

FIG. 5(A) illustrates the executions on an exemplary 25 that host necd not bc an active host - 
three-hop path for the above-described conventional tracer- FIG. 6 shows an example of a router identifier report 
oute program. FIGS. 5(B) and 5(C) illustrate the executions generated by the implementation of the above-described 
on the exemplary three-hop path for a collect-en-route report-en -route program, executed on the six-node active 
program and a report-en-route program, respectively, in network of FIG. 3. Note that the reports from the FreeBSD 
accordance with the invention. These programs provide machines are truncated due to an implementation problem of 
different options for gathering information along a given the SNMP agent in FreeBSD. The active packet that gen- 
path between two network nodes, and each optimizes a erates this report can be sent from any host as long as its path 
different objective function. Relevant parameters include goes through an active node. The first active node, tishrey in 
communication cost, e.g., the number of hops required for a this example, diverts the active packet to its active engine, 
given message, and the amount of time required to perform as the packet uses the well-known active port number 
a particular task. (3322). The packet contains the class file of the Java code 

As shown in FIG, 5(B), the colkcKn-route program shown in FIG. 7 as well as 9 bytes of data, which contain the 
sends a single packet that will traverse the route and collect rc P ort destination IP address, the IP address of the destina- 
the desired information from each active node. When the 40 ,ion end-point of the path, and a hop count, 
packet arrives at the destination node, it sends the data back As the session number of this packet does not match any 
to the source, or to any suitable management station. This existing session in this node, a new session will be created 
design minimizes the communication cost since a single using the Java code in the active packet. The packet itself is 
packet is traveling along each link in each direction. then delivered to this session as the first packet. The session 

As shown in FIG. 5(C), the report -en-route program sends 45 rcads the data from the capsule, generates a copy of the 
a single packet along the path. When the packet arrives at a active packet to be sent towards a destination, sends a report 
node, it sends the required information back to the source nome > and terminates. The generated copy is then inter- 
and forwards itself to the next hop. This design minimizes cepted by the next node on the route to the destination in 
the time of arrival of each part of the route information, at which exactly the same scenario repeats itself. The reports 
the expense of increased communication cost. 50 arc scnt . to thc destination specified in the code (the report 

destination may be part of the data carried in the capsule), 
TABLE 1 which can be different from the host that originated the 

application. 

In the Java code of FIG. 7, session is a new instance of the 
55 class Act. The constructor takes -9 as an argument that 
indicates the number of data bytes in the capsule. The 
program and the data are then retrieved using Act methods 
described herein. A new active packet with the appropriate 
hop count is then prepared, and sent to the destination 
TABLE 1 compares the performance of the three different 60 address. A report is then generated. Local information from 
algorithms illustrated in FIGS. 5(A), 5(B) and 5(C). In this the router is gathered using the SNMP interface. Although 
table, the communication cost is measured in messages tms embodiment uses a full M1B specification of the 
times number of hops, and the time is measured in hop requested values, in other embodiments part of this interface 
count. The time of data arrival assumes that the nodes along mav oe overridden by a different Java interface to retrieve 
a given route are numbered 0, 1, 2, ... n. The conventional 65 som e of the most important information, 
traceroute program has time and communication complexi- The following is a description of a number of functions 

ties that are quadratic in the path length, as is apparent from associated with the class Act: 







Time of Data 


Algorithm Used 


Communication Cost 


Arrival from Node i 


trace route 


n(n + 1) 


i(i + 1) 


collect-en- route 


2n 


2n 


rcport-cn-routc 


n(n + 3)/2 


2i 
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Aia (mi leu) — A constructor which opens a datagram group of receivers wouJd generally be implemented by 

socket, and sends an op -code to the manager adminis- either sending a unicast message to each receiver, or by 

tration socket to announce that the session is active. broadcasting the message to the entire network. 

The opened socket is used for all the outgoing com- I* ^ assumed in this example that a message is comprised 

muni cation to the manager. The constructor saves the 5 °£ 3 header with a list of receivers, and a body which, for a 

program code and the initial data. The field len is lar S e g rou P of receivers, is much smaller than the header, 

optional and it indicates the amount of initial data One may make use of the fact that the union of all the routes 

carried with the program packet. The program packet is from lhe . °. ri ginator to the receivers is a directed tree rooted 

also referred to herein as a capsule. al * e originator. This tree is referred to herein as a dissemi- 

L1 . , A n A „ A iL , r , n nation tree. For purposes of simplification and clarity of 

public byte[ ] ge Prog( )-Retums the program code from io illustration> it wiu be assumed disseminalion tre ' e is 

the capsule. Used by a session to send itself. a binary balanced tree with the rec€ ivers at the leaves, as 

public byte[ ] ge Unit Vans ( ) — Returns the initial data shown in FIG. 9. 

from the capsule. The active network approach to the message dissemina- 

public byte[ ] rec ( ) — Used to receive a packet without tion application involves partitioning the receiver list at the 

the active header. 15 source according to the first hop on the path to each receiver, 

public byte[ ] recraw( )— Used to receive a packet with ^ partitioning is continued at every intermediate node 

the active header. This method may be useful when the unUl } he messa § e a ™ es at the tree leaves * In ^J? manne /> 

header information is required, e.g., for debugging. *™ 1 T T v ^age traverses each hnk in the 

^ ; & dissemination tree. For a balanced binary tree with n leaves, 

public void send (byte[ ] m, InetAddress destAddress)— 20 the message complexity is 2n, while the above-noted unicast 

Used to send a packet m to address destAddress. The solution has a message complexity of n log n. 

ANEP header and above-described default option are In each active node> a po Ssibly large lisl of addresses may 

generated by the manager and are not part of the packet need t0 be partitioned. This requires more processing at each 

m - node than in the bottleneck detection example. In fact, the 

public void killme ( ) — Used to signal the manager that 25 processing time will be linear in the receiver list length, 

the session terminated and can be cleaned. since the next hop of every receiver needs to be checked, 

public void keepme( ) — Used to signal the manager that This requires access to the routing table at the router, which 

the session is still alive. is provided in the architecture described above. 

As previously noted, the invention is not limited to The time it takes the algorithm to deliver the message to 

collecting only node identifiers. Other embodiments could, 30 all the destinations depends both on the delay along the 

e.g., check one or more IP counters in a router, instead of or links, and the delay due to the processing in the nodes. The 

in addition to the router name. The change in the code to link delay contribution is the same for both solutions since 

implement such a change is minimal, i.e., one need only the IP routing is not changed. If the message is distributed 

request a different MIB variable. For example, a request for using the unicast approach, O(n) processing cycles are 

.1.3.6.1.2.1.4.6.0, which is an MIB variable counting the 35 required at the sender. On the other hand, the above - 

number of IP packets forwarded by the router, results in a described active network approach partitions the address list 

report such as the one shown in FIG. 8. Note that the reports in each intermediate node. The delay due to this active 

in the examples of FIGS. 7 and 8 are received at inbar, which network approach is about twice that of the unicast 

is not an active node. approach, since at every level of the tree the address list is 

It can be seen from FIG. 8 that the number of forwarded 40 halved. Nonetheless, the address dissemination example 

packets is increased between the two executions. The serves to illustrate the manner in which an active network 

counter value is 0 for the last router, as currently only one approach can be used to trade off delay and network utili- 

of its interfaces is connected, and thus it does not forward zation. In this example, the active network approach 

any packets. Note also that the reports may arrive out of achieves a logarithmic improvement in utilization at the cost 

order due to the difference in response time between the 45 of only a constant factor in delay. 

SNMP servers in the machines. It is the responsibility of the The above-described active network approach may be 

application GUI to display the reported information in a used in numerous other applications, such as, e.g., adaptive 

format convenient for the user. control, router configuration, element detection, and net- 

The other example application to be described herein is work mapping. The active network approach is also well 

message dissemination. In many network management 50 suited for use in security management applications such as, 

applications there is a need to deliver a message to an ad-hoc e.g., intruder detection, fighting denial of service attacks, 

group of machines. For example, using an autoconfiguration etc. The active network approach can also be used to support 

application, a group of routers might need to be reconfigured solutions to other problems that are not necessarily consid- 

due to a change in the network. As another example, a ered part of network management, such as, e.g., search 

monitoring application may periodically query all the hosts 55 mechanisms, smart mail, multicasting, hop-to-hop flow 

it did not hear from in the last period. In yet another control, etc. 

example, a security application might collect information As noted previously, the present invention provides an 
from a group of routers based on the attack pattern it active network approach to distributed network management 
suspects. that simplifies deployment of distributed network manage- 
In these and other similar applications, the machine group 60 ment applications. The approach provides a number of 
is ad -hoc defined for the purpose of dissemination of a single advantages over conventional network management 
message, in contrast to being defined as a long-lasting group approaches, including the following advantages: 
as in multicast applications. Since the group is defined by the 1 . Generality and simplicity. The active network approach 
recipient list of a single message, it is not efficient to form of the invention is not be limited to one language, and 
a multicast group or to invest in any other long-term 65 supports languages that are in general use. A given node of 
infrastructure. Without the active network approach the network is general enough to support both long-terra and 
described herein, such a message dissemination to a large short-term applications. 
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2. modularity, in accordance wiih ihe invention, a net- 3. ine apparatus of claim 2 wherein each of at least a 
work node is separated into router and active engine mod- subset of the network nodes comprises a computer. 

ules with a clearly-defined interface between them. More 4. The apparatus of claim 2 wherein at a given one of the 

particularly, in the illustrative embodiment, the forwarding netW ork nodes the router and the processing engine reside 

mechanism of a router is separated from the operating 5 on the same machine 

environment where the active packets are executed. The 5 ^ apparatus of claim 2 wherein at a given one of the 

l e ;f well-accepted standards, such as Java, nelwork nodes ±Q ^ ^ { ^ sj ine reside 

SNMP. and the above-noted active network encapsulation « u- 

\ ✓ 4 »tt-^\ t i ™ t • on ditterent machines. 

protocol (ANEP), as the API in which the router and active ^™ . * 1 ■ i »_ ■ i_ 

r , j 1 u • r 4- 6. The apparatus of claim 1 wherein the processing engine 

engine modules exchange information. 10 , , r - . 7 

3. Interoperability and heterogeneity. In many ^f 5 "P"! of ™ MOn ?' . eac u h corresponding to a 
applications, active nodes will co-exist with non-active distributed task to be performed in the network. 

node. Furthermore, incremental deployment of active nodes 7 * ^ apparatus of claim 6 wherein each of the sessions 

with co-existing conventional routers is a natural evolution has associated therewith a unique network identifier, such 

path. In such a scenario, it is unlikely that an application 15 that differcclt programs on different network nodes can 

running on an active node could explicitly know the belong to the same session. 

addresses of its other active neighbors. To this end, the 8. The apparatus of claim 1 wherein the router supports a 

invention supports blind addressing, in which the active blind addressing mode which is used to send designated 

node need not know the specific address or the location of packets from the network node via unspecified network 

other active nodes. 20 nodes in a particular direction toward a destination node, 

4. Application life span. In many network management such that a router in a first unspecified node along a route to 
applications, e.g., monitoring and billing, there is a natural the destination node intercepts the packet and sends it to an 
need for an application to reside in a node for a long period associated processing engine of that node. 

of time. The active network approach of the invention 9, The apparatus of claim 1 wherein the network man- 
supports such applications, as well as tasks having a short 25 agernent function comprises a program that sends at least 
life span. onc p ac fc e t to traverse a route through the network from a 

5. Network layer interface. The active network approach source node l0 a destination nodc and spcc j ne d 
allows an application to have easy and standard access to the MoTm;itiou frora each node along the route, wherein when 
local information at a node, since in many applications the ^ ket afrives a( ±e destination nod it ^ n6s the 
action taken by the packet depends on this information. Tins 30 ified infonnation back t0 lbe source node . 

access supports read and write operations, thereby allowing ~. ♦ * 1 • 1 u ■ .u * 1 

the application to take corrective actions. 10 ^ apparatus of claim 1 wherein Oe network man- 

6. Cost visibility. The active network approach of the *& mcn \ composes a program that sends at least 
invention allows applications to be aware of costs, both in one P acket t0 traverse a rcmte throu S h the network from a 
terms of node resources such as CPU, memory, etc., and in 35 source node t0 a destinatlon node and collect specified 
terms of global network resources such as bandwidth and information from each node along the route, wherein when 
delay. Advanced distributed tools, such as CORBAand Java the packet arrives at a given node along the route, it sends 
RMl, which in general hide much of the actual costs from the specified information back to the source node, and 
the user, are not required. forwards itself to the next node along the route. 

7. Safety and security. The invention supports additional 40 H The apparatus of claim 1 wherein the network man- 
functionality without affecting legacy network operation, agement function comprises a message dissemination appli- 
and can be configured to prevent unauthorized network cation in which a message is to be directed from a source 
management applications from affecting any other applica- node to each of a plurality of destination nodes, such that the 
lion. union of all the routes from the source node to the destina- 

As previously noted, the above -described embodiments of 45 tion nodes may be represented as a directed tree rooted at the 

the invention are illustrative only. Alternative embodiments source node. 

may be implemented in other types of computer networks, 12 -phe apparatus of claim 11 wherein a list of the 

using hardware and software configurations other than those destination nodes is partitioned at the source node in accor- 

specifically described herein These and numerous other dance ^ a fifSt h 0Q ^ h tQ each of ^ destiDation 

a ternative embodiments within the scope of the following 50 nod and ^ rtitioni is continucd at each intermediate 

claims will be readily apparent to those skilled in the art. , . 4 *, , , t . , 4 , 

nn _ A . t . , . node between the source and destination nodes until the 
Wrist is clflimcfi is* 

- t ' - ■ , . . , .* , . message arrives at each of the destination nodes, such that 

1. An apparatus for use in implementing a distributed . 1 c . . , . 

. ™ c • • * 1 »u . a single copy of the message traverses each link in the 

network management function in a network, the apparatus directed tree 

comprising. , 55 13. A method for implementing a distributed network 

a network node having associated therewith a router and management ^ nc{ion in a network , the met hod comprising 

a processing engine, wherein the router is operative to ^ steps of* 

divert designated packets relating to the distributed „ 

network management function to the processing configuring a network node to include a router and a 

engine, and the processing engine supports one or more 60 processing engine, 

sessions, based at least in part on the designated diverting designated packets relating to the distributed 

packets, for implementing at least a portion of the network management function from the router to the 

network management function. processing engine; and 

2. The apparatus of claim 1 wherein the network includes maintaining one or more sessions in the processing 
a plurality of network nodes, with each of at least a subset 65 engine, based at least in part on the designated packets, 
of the network nodes including the router and processing for implementing at least a portion of the network 
engine. management function. 
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14. An arucie oi manutacture comprising a macnine- 
readable storage medium for storing one or more software 
programs for implementing a distributed network manage- 
ment function in a network, wherein the one or more 
software programs when executed implement the steps of: 
configuring a node of the network to include a router and 

a processing engine; 
diverting designated packets relating to the distributed 
network management function from the router to the 
processing engine; and 
maintaining one or more sessions in the processing 
engine, based at least in part on the designated packets, 
for implementing at least a portion of the network 
management function. 
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15. A network comprising: 

a plurality of nodes, each of at least a subset of the nodes 
including a router and a processing engine, wherein the 
router in a given one of the nodes is operative to divert 
designated packets relating to a distributed network 
management function to the processing engine of the 
given node, and the processing engine supports one or 
more sessions, based at least in part on the designated 
packets, for implementing at least a portion of the 
network management function in the given node. 
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